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_ SUMMARY 


/ 

The automation of low-altitude rotorcraft flight depends on the ability to detect, locate, and navigate 
around obstacles lying in the rotorcraft’s intended flightpath. Computer visjon techniques provide a pas- 
sive method of obstacle detection and range estimation, for obstacle avoidance. Several algorithms based 
on computer vision methods have been developed for this purpose using laboratory data; however, fur- 
ther development and validation of candidate algorithms require data collected from rotorcraft flight. A 
data base containing low-altitude imagery augmented with the rotorcraft^ and sensor parameters required 
for passive range estimation is not readily available. This paper focuses on the methodology used to de- 
velop such a data base from flight-test data consisting of imagery, rotorcraft and sensor parameters, and 
ground-truth range measurements. As part of the data preparation, a technique for obtaining the sensor 
calibration parameters is described. The data base will enable the further development of algorithms for 
computer vision-based obstacle detection and passive range estimation, as well as provide a benchmark 
for \^nficatiomof range estimates against ground-truth measurements. 

INTRODUCTION 

To minimize the risk of detection in high-threat environments, rotorcraft fly at low altitudes so the 
pilots can use natural terrain, vegetation, and manmade structures to mask the rotorcraft’s presence. As 
the rotorcraft approaches the terrain, the degree of concealment increases; however, pilot workload also 
increases because the same objects that provide concealment constitute an obstacle course that must be 
negotiated by the pilot. Figure 1 depicts three modes of low-altitude rotorcraft flight. The most effective 
and complex of these is nap-of-the-Earth (NOE) flight, during which the rotorcraft executes both lateral 
and vertical maneuvers below tree top level. The obstacle avoidance task during NOE flight is demanding, 
leaving the pilot little opportunity to focus on mission-related activities. Mission effectiveness can be 
increased by developing automation tools to assist the pilot in obstacle avoidance, thus enabling greater 
attention to be focused on meeting the mission objectives (ref. 1). 

The automation tools must be able to perform obstacle detection and range estimation in order to 
collect information about the surrounding environment. Operationally, these requirements will likely be 
met through a combination of active sensors, such as millimeter-wavelength radar or laser range scanners, 
and passive sensors, such as forward-looking infrared (FLIR) imaging systems or low-light-level television 
(LLLTV) cameras. Reliance on passive sensors whenever feasible, however, will minimize the risk of 
detection, so it is of interest to determine the degree to which obstacle detection requirements can be met 
solely through the use of passive sensors. If the detection requirements cannot be met using passive sensors 
alone, supplementary data can be acquired by the minimal use of active sensors directed selectively, based 
on the passive sensor information. 

NASA, in cooperation with the U.S. Army, has initiated a program to determine the extent to which 
passive range estimation can be used as a basis for obstacle avoidance. Several methods have been devel- 
oped to perform passive range estimation (refs. 2-6), and early algorithms have been tested on laboratory 
data with promising results. Further development of passive range estimation algorithms for use in obsta- 
cle avoidance during NOE flight requires data collected through actual rotorcraft flight, including not only 
the measurements required to perform passive range estimation but also measurements of true range for 
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Figure 1. Three modes of low-altitude rotorcraft flight. 


validation. The purposes of this paper are to present a methodology used to develop the required data set 
and to describe the resulting data set. 

One method of obstacle detection and range estimation is based on computer vision techniques that 
operate on a sequence of images. This paper focuses on the analysis of video-type imagery, but the same 
techniques also apply to other types of imagery, such as FLIR, which could lead to a nighttime-capable 
system. These techniques are based on the well-defined relationships among the sensor motion, the ap- 
parent motion over time of obstacles in the sensor imagery, and the position of the obstacles relative to 
the sensor. We assume that sensor motion is known through measurements made by an inertial navigation 
system onboard the rotorcraft; we are interested only in determining the locations of obstacles. 

This paper first provides a general description of the passive ranging concept, for the purpose of defin- 
ing the data requirements. Descriptions follow of the flight experiment, the procedures used in processing 
the flight data, and the experimental method developed to determine the characteristic parameters of the 
sensor. A general discussion of the resulting data set and plans for future activities completes the paper. 

The author wishes to acknowledge his colleague Ray Suorsa for his valuable assistance and sugges- 
tions throughout flight test and sensor calibration. 

PASSIVE RANGING CONCEPTS 

Consider a rotorcraft- mounted sensor that is in motion with respect to an inertial, Earth-fixed frame of 
reference while it observes an obstacle P whose location is fixed in the Earth frame, as shown in figure 2. 
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Figure 2. Passive ranging geometry. 


We wish to determine the relative position of the obstacle P with respect to the sensor, 

P = r p — r 3 


( 1 ) 


The imaging sensor maps the obstacle P, whose location in sensor coordinates is p a = [p %a , p Vs , p z J r , onto 
the image plane at Q by perspective projection, as follows: 

( 2 ) 


~u 


/ px 3 / pz a 



-fPy./PzA 


where / is the focal length, a characteristic of the sensor. 

As the sensor moves, p s changes, so the location of the image point Q changes according to 


= j_ r fp x . - up Za 

Pz, L f Pv> ~~ V P*» J 


(3) 


Because the sensor frame is moving, the derivative of p a is determined using the Coriolis equation 

Pt = P» + w x p (4) 


where p e is the derivative of p in the Earth frame and is equal to the negative of the sensor velocity in the 
Earth frame, p , is the derivative of p in the sensor frame, and w is the rotation of the sensor frame relative 
to the Earth frame. Let V, = [V x „V v „V z ,] T and w 3 = [w,„ u Va , u Zl ] T be the linear and angular velocities 
of the sensor with respect to the Earth frame and resolved into sensor coordinates. Then, noting that P is 
fixed in the Earth frame, we use equations (1) and (4) to obtain 


p a = -V a - uj a x p a 


(5) 


The motion of the image point Q can now be written in terms of the sensor motion using equations (3) 
and (5): 


3 



(6a) 

(6b) 


U = Uy + 1 
V — \>t + Vft 

where the image-point motion has been separated into components resulting from the translational and 
rotational motion of the sensor, denoted by the subscripts T and R. The components are defined as 


u T = (~fV x , + uV z ,) fp t . 

(6c) 

V T = {-fVy, + VVz.) fpz. 

m 

= jWx, - / (l + /!-) Wj. + vw*. 

(6e) 

VR = f (l + jr) w I( - JU V , + uw*. 

(60 


This motion of the image point Q caused by sensor motion is known as optical flow. It is assumed that V, 
and w, can be derived from the rotorcraft’s inertial navigation system; this will be addressed in the next 
section. If the sensor motion, the focal length, and the optical flow are known, the range, p lt , of the object 
P corresponding to the image point Q can be determined from the optical flow equations (eqs. (6)). The 
full vector p a can then be recovered by using the perspective projection equation (eq. (2)). 

DATA REQUIREMENTS 

In order to test passive range estimation algorithms, it is necessary to have measurements of the sensor 
motion, V, and w 4 ; knowledge of the focal length, /; and a sequence of images in which the location of 
any image point can be determined in [u, v] coordinates. The optical flow information [ti, v] is extracted 
from the image sequence by tracking over time the location of the image point Q corresponding to an 
obstacle P. This completes the set of data required to perform passive range estimation. Measurements of 
the actual range are also necessary for comparison with the results of the ranging algorithms. 

The sensor motion can be determined from the rotorcraft’s velocity, Vi, and angular rates, oj b , both 
measured in the rotorcraft body frame of reference. Using the Coriolis equation and the well known coor- 
dinate transformation equations, we have 


Va = TbaiVb + Wi X Pba) 

= TbaUb ( 7 ) 

where p^ is the vector extending from the body origin to the sensor origin and is the direction cosine 
matrix, defined by three Euler angles (see table 1), that specifies the orientation of the sensor frame relative 
to the body frame. The three-element position vector p b3 and the three Euler angles that define T bs are 
known as the external camera parameters. 
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Table 1. Definition of direction cosine matrix 


Tbs (V’ > 0 > <f>) ~ 


0 10 ' 
0 0 1 
1 0 0 


r n Tn Tn 
Tzi T22 T23 
T3I T32 T33 


T11 = cipc9 
Ti3 = —s9 

T 22 = sips6s(j> + cipc(f> 
T31 = c\ps6c<p + s\ps<p 
T33 = c6aj> 


T12 = sipc9 

T21 = cips9s<p — sipc<p 

T23 = c9s(p 

Tn = sips9c(p — cipscp 


Note that the sensor frame described by p bs and T bi is not defined by any external physical features of 
the camera, but reflects the location of the focal point and the orientation of the image plane, as illustrated 
in figure 3. The image plane is formed by a 1/4-inch-square, electronic charge-coupled device (CCD) array 
located inside the camera. The x, and y s sensor axes are parallel to the image-plane axes [ u, v] and are 
oriented along the rows and columns of the CCD array. The z 3 axis lies along the principal ray and is 
orthogonal to the image plane. The origin of the sensor frame is located at the camera focal point. 

The location of an image point in [u, v] coordinates can be determined from the digitized images 
according to the relationships 

it = (riu — n^, ) Su/oc 

v - {n^ - n vo ) 8v (8) 



Figure 3. Sensor model. 
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where [n^n,, ] are the measured row-column indices of a picture element (pixel) in the digitized image 
array; fa*, , ] are the row-column indices of the piercing point, where the principal ray intersects the 

digitized image plane; [6u,6v] are the physical spacing of the CCD array elements (available from the 
camera manufacturer); and a is a scaling parameter, introduced by the digitization process, that defines the 
mapping between the CCD array elements in the camera and the pixels in the digitized image array. The 
four parameters n tlo ,n Vo , a , and /, the focal length, are known as the internal camera parameters. 

To understand the need for the scaling parameter, a, it is necessary to understand the method used to 
digitize an image generated by the CCD array sensor. Once the CCD array has been exposed, the intensity 
values registered by the array elements are read out row by row, producing a video signal in the form of an 
analog voltage waveform. This analog video signal must be digitized so the image can be processed by a 
computer. A timing mark in the video signal at the beginning of each new row ensures the preservation of 
the vertical dimension of the image during digitization; however, if the digitization sampling rate differs 
from the rate at which the CCD array elements of a row are read out to form the video signal, then the 
horizontal dimension corresponding to a digitized pixel will differ from that of a CCD array element by a 
scale factor a. For a typical 512 x 512 pixel image, a small discrepancy in sampling rate can lead to a 
significant error in the u-coordinate of a feature located near the edge of the image. 

Figure 4 summarizes the data and analysis required to validate passive range estimation algorithms. 
The data set comprises digitized imagery data in the [n^, ] coordinate system, the results of an extensive 

camera calibration to provide the six external and four internal camera parameters, rotorcraft motion state 
data, and measurements of actual rotorcraft position relative to the obstacles. The development of the 
data set is considered in two sections. First is the acquisition of the flight data, including imagery data, 
rotorcraft state measurements, and radar tracking data for true range information, all of which must be time 
correlated. Second is the calibration of the imaging sensor, in this case a video camera. 

FLIGHT DATA COLLECTION 

The flight experiment designed to collect the necessary data is depicted in figure 5. The sensor plat- 
form was a CH-47B Chinook helicopter, onboard which a video collection and recording system was 
installed for the acquisition of the imagery data. A Cohu 6400-series video camera was rigidly mounted 
under the rotorcraft nose and oriented roughly along the direction of flight so as to observe designated 
obstacles that the rotorcraft would encounter. The camera’s position and orientation with respect to the 
rotorcraft was held constant throughout the flight project. 

To determine the roiorcraft motion states, the following measurements were acquired by instrumen- 
tation (listed in parentheses) onboard the CH-47 and telemetered to the ground station at the test-flight 
facility for recording: (1) longitudinal and lateral inertial velocities (Doppler radar), (2) linear accelera- 
tions (accelerometers), (3) Euler angles (inertial navigation system), and (4) angular rates (rate gyros). 

To ensure that the derived rotorcraft velocities and rotational rates were as accurate as possible, these 
data were processed using state estimation algorithm referred to as SMACK (Bach, R. E., Jr.: State Es- 
timation Applications in Aircraft Flight-Data Analysis (A User’s Manual for SMACK), to be published 
as NASA Ames Reference Publication). This algorithm uses the well-known set of rigid-body dynami- 
cal equations of motion as a model, processing the measurements together in an “optimal” way to ensure 
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Figure 4. Major elements of the data set. 


the internal consistency of measured states, improve the knowledge of poorly measured or unmeasured 
states, identify instrument-bias errors and scale-factor errors, and estimate the states during brief periods 
of telemetry dropout. In this way, a best estimate of the rotorcraft’s position, orientation, velocity, and an- 
gular rates was obtained, based on all available measurements. It is important to note that state estimation 
is used to develop a high-precision, internally consistent data set for research purposes. An operational 

sytstem for passive range estimation would acquire rotorcraft state information directly from onboard sen- 
sors. 


True range measurements were obtained by a two-step process using a laser tracker. First, the laser 
tracker measured the position of the rotorcraft throughout each test flight. The resulting data was recorded 
at the ground station on a common time base with the telemetry data. Second, at the completion of a test 

flight, the laser was used to measure the position of the (stationary) objects that served as the designated 
obstacles. 

To reference the imagery data with the rotorcraft state data and the true range measurements, a time 
source onboard the CH-47 was synchronized with the time source at the ground station to 1-msec accuracy. 
Subsequently, a message containing the current time was overlaid on the upper left-hand comer of each 
videoimage. 
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Figure 5. Flight experiment overview. 


To simulate typical NOE maneuvers, test flights were conducted within about 40 ft of the ground at 
speeds not exceeding 40 knots. Typical flight profiles were straight-and-level flight, banked- and-curved 
flight, pedal turns, bob-ups, and transitions to and from hover. In addition, obstacles were chosen to repre- 
sent varying levels of difficulty of detection; land vehicles, buildings, transmission towers, natural terrain, 
and vegetation were used. 


CAMERA CALIBRATION 

It is difficult to directly measure the values of any camera parameters to a sufficiently high degree 
of accuracy, so the values of the 10 unknown parameters must be determined experimentally. This is 
possible because the external parameters (p bs and T ba ) define how the location of a point P known in body 
coordinates, p b , is transformed into sensor coordinates, 

p 3 = T ba (p b - p b3 ) (9) 

and the internal parameters (tj„ 0 , n Vo , a, and /) define how the point is subsequently mapped from the 
sensor frame to some measurable pixel location in a digitized image produced by the sensor, 

n» = + af(p x jpj/6u 

+ / ( PJPz .) /5v (10) 

Thus, measurement of object locations in the body frame and measurement of the corresponding pixel 
locations in the digitized image provide the necessary data for determining the unknown parameters. Note 
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that the data should exhibit variation in p z< in order to permit robust separation of the focal length, /, from 
the component of p bs along the principal ray during parameter estimation. Substitution of equation (9) into 
equation (10) yields 2 nonlinear equations in the 10 unknown parameters, so a minimum of 5 measurements 
of p b and [ rtu, riv ] are required to uniquely determine the parameter values; however, because the emphasis 
here is on accuracy, it is desirable to have redundant measurements and to use a least squares method to 
solve the equations. 

To acquire the necessary data, a calibration grid was constructed to provide known calibration points 
P that would represent obstacles. Figure 6 shows the experimental setup for collecting the calibration 
data. The rotorcraft was placed at an arbitrary location on the ground with the calibration gnd positioned 
in the sensor field of view. The calibration points were the intersections of the horizontal and vertical grid 
lines. The first objective was to measure the location of the calibration points with respect to the rotorcraft 
body axes, but the position and orientation of the body frame were unknown. The rotorcraft manufacturer 
provided the locations in body coordinates of distinctive, accurately known, external rotorcraft features. An 
arbitrary, user-defined coordinate system was then established at the location of a transit and laser distance- 
measuring equipment (DME); this allowed range, bearing, and elevation measurements to be made of the 
external rotorcraft features. Solution of an over-determined triangulation problem yielded the location and 
orientation of the body frame relative to the user frame. The same transit and laser distance-measuring 
equipment were then used to measure the locations of the calibration points on the calibration grid with 
respect to the user axes. Finally, knowing the locations of the calibration points in user coordinates and the 
relationship between user and body frames, the positions of the calibration points in body coordinates were 
determined. This procedure was repeated with two different user coordinate systems to improve accuracy 
and confidence in the results. 

The pixel location corresponding to each calibration point was determined manually from the image 
generated by the sensor. To measure the pixel locations of the calibration points to sub-pixel accuracy, a 
calibration grid arrangement, rather than isolated calibration points, was used, in the following manner. 
The image intensity profile perpendicular to a grid line was viewed and the peak of the profile interpolated 
to obtain, to sub-pixel accuracy, a point on the center of the line. A series of these points were then used to 



Figure 6. Data collection setup for camera calibration. 
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reconstruct the true line using a least-squares line-fitting algorithm. The intersection of two lines identified 
in this manner determined the location of a calibration point to sub-pixel accuracy. The entire calibration 
process was repeated with the calibration grid in a different location to ensure sufficient variation in p Zt . 

A method was developed that uses a Newton-Raphson technique to solve the nonlinear least-squares 
problem to determine the unknown parameters. The method is based on the work of Tsai (ref. 7) but has 
additional provisions to allow estimation of the piercing point location, [n„ 0 , n„ 0 ] . The cost function to be 
minimized contains penalty terms for the discrepancy between the measured and predicted image plane 
location [ i\, n„ ] of each calibration point and for violation of the radial alignment constraint (RAC). The 
RAC states that the vector n from the piercing point to an image point Q must be parallel to r 2 , the vector 
component of p, parallel to the image plane (see fig. 3). A more detailed discussion of the parameter 
estimation technique is beyond the scope of this paper; however, the parameter estimates obtained are 
given in table 2. 


Table 2. Camera calibration parameters 


Parameter 

Estimated 

value 0 

Uncertainty 
of estimate 

Units 

Pbs x 

275.4 

0.5 

in. 

Pbs y 

-12.5 

0.2 

in. 

Pbs , 

83.3 

0.2 

in. 

1>bs 

0.3 

0.2 

deg 

Ob, 

-8.0 

0.2 

deg 

fobs 

-0.4 

0.1 

deg 

^0 

253.3 

2.4 

pixels 

"vo 

238.3 

1.6 

pixels 

a 

1.005 

0.001 

none 

f 

0.242 

0.001 

in. 

Su 

3 .89 x 10 - 4 

n/a 

in. 

Sv 

3.89 x IQ" 4 

n/a 

in. 


a based on 40 calibration points 
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RESULTS 


In final form, ready for use in passive range estimation research, the data base contains time-correlated 
measurements of 

1. imagery data 

2. sensor velocity and angular rates, in sensor coordinates 

3. sensor internal calibration parameters 

4. sensor external calibration parameters 

5. sensor position and orientation, in Earth coordinates 

6. obstacle locations, in Earth coordinates 

The first three data elements are required for performing passive range estimation. The fourth element 
allows range estimates in sensor coordinates to be expressed in terms of the rotorcraft body frame so that 
obstacle information can be displayed to the pilot or used by the flight control system. The final two 
data elements are used to validate the range estimates obtained from a passive range estimation algorithm. 
Additionally, since the sensor position and orientation histories are known, a global model can be developed 
that includes all obstacles encountered by the rotorcraft throughout its flight. 

Figure 7 shows one image collected during a flight test, with the associated sensor states and true- 
range data. In the flight-test scenario depicted, several vehicles were stationed along a runway ramp to 
serve as obstacles around which the rotorcraft maneuvered. Apparent in the image are the time stamp and 
several features, each of whose true range has been measured. The sensor calibration parameters (which 
are time invariant) are those given in table 2. 


11 


ORIGINAL PAGE 

BLACK AND WHITE PHOTOGRAPH 



V s = [0.6, -1 8,38.6] t ft /sec 
= [— 0.19, — 0.39, 0.44 ] T deg/sec 


Object 

Px$ 

(ft) 

Pv. 

(ft) 

Pz. 

(ft) 

A 

-66 

1 

184 

B 

-65 

-15 

435 

C 

69 

-31 

580 

D 

68 

-19 

561 

E 

69 

-8 

334 

F 

69 

-3 

312 


Figure 7. Sample data from a flight test. 
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CONCLUDING REMARKS 


A complete “real world” data base has been developed from data acquired through rotorcraft test 
flight, to enable further research into the use of passive range estimation for obstacle avoidance during 
NOE flight. The data base has been designed to incorporate measurements required by a wide variety 
of single-sensor, passive range estimation algorithms, so as to serve as a benchmark for comparison of 
different methods. During the data preparation, a camera calibration methodology was developed to obtain 
accurate knowledge of the sensor’s characteristic parameters. The camera calibration technique as well as 
the base can potentially be applied to research in robotics and the autonomous navigation of vehicles. 

Future directions include a comparative study, based on the data set described here, of current passive 
range estimation algorithms followed by further research on the most promising algorithms. There is also 
interest in applying the experience gained in developing the single-camera data base to the production of 
a similar data base to support research in multicamera (stereoscopic) methods of passive range estimation. 

To obtain the images in digitized form, along with the sensor states, sensor calibration parameters, 
and true range measurements, researchers in passive range estimation may contact the author. 
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